1. Field of the Invention
The present invention relates to an image processing apparatus for accurately extracting one or a plurality of objects utilizing a thresholded differential image processing technique when a plurality of stationary objects and a plurality of moving objects are contained together in an image of a time sequence of images.
More specifically, the present invention relates to an image processing apparatus, which allows both a background image and an image including at least one stationary object or at least one moving object (having a speed not more than a predetermined speed) to be extracted, and also allows a difference-calculation process to be carried out between the images.
In such an image processing system, it is possible to distinguish a stationary object or a moving object in the images and it is also possible to analyze the movement of each of the moving objects.
Further, the present invention relates to an image processing apparatus, in which a plurality of markers are provided in a background where the objects move, and these markers are extracted by utilizing an image processing technique similar to the case in which the moving objects are extracted, and further it is discriminated whether or not the thus extracted markers are in the steady state.
If the markers are in the steady state, portions where the moving objects and the markers overlap each other can be determined. Therefore, the number of the markers (the size of markers displayed in each image), which are in the steady state and exist between two moving objects, can be calculated to obtain a distance between two moving objects.
In general, supervisory systems using the above-mentioned image processing technique can be utilized in various places. Each of these supervisory systems serves to rapidly locate an accident, a disaster, and the like. Recently, such supervisory systems are likely to be utilized for preventing such accidents, disasters, and the like, in addition to a function of merely detecting the existence of an accident, etc.
To meet this need, it is necessary to extract or identify an object which moves with an abnormal motion that will cause such an accident, a disaster, and the like. Therefore, an efficient technique is needed for rapidly and accurately detecting a moving object which demonstrates such an abnormal motion.
More specifically, it is required for the supervisory system to detect and analyze the movement of each of a plurality of moving objects contained in a series of images. Further, it is also necessary for the supervisory system to rapidly calculate a distance between the two moving objects with a high degree of accuracy.
2. Description of the Related Art
Some techniques for analyzing the movement of each of a plurality of moving objects by utilizing an image processing apparatus are typically disclosed in Japanese Unexamined Patent Publication (Kokai) No. 5-159057 and No. 5-159058.
In each of these techniques, first, regions where moving objects may be positioned are extracted using a predetermined assumption. Next, a specified moving object is distinguished from the other objects, on the basis of various characteristics, e.g., the size of each of the regions, and the central position of each region. Subsequently, in accordance with a change of the position of the moving object with a lapse of time, the movement of the moving object can be analyzed.
For example, when an analysis of the motion of a man is to be performed, a given portion of an image which is to be analyzed, is extracted from the image. Next, with respect to the extracted portion, i.e., an object to be processed, various characteristics, e.g., the position of projections and the location of central positions, are calculated, and used to distinguish the object from the other portions. Further, the process is executed with respect to a plurality of images in a time series, i.e., continuous motion type images.
According to the above-mentioned technique, to ensure obtaining adequate attributes, e.g., a speed of the object, it is necessary to analyze all the areas where the same original object can exist in the time series, and to identify the objects as the same original object.
More specifically, if a plurality of objects respectively existing in a plurality of the time series images are not accurately correlated with each other, by analyzing all the areas where the object can exist, it is difficult to calculate the speed of the object with a sufficiently high accuracy.
In the case where only one moving object exists, a process for correlating a plurality of objects in the continuous images with each other is relatively simple. In this case, it is possible to easily obtain the attributes, e.g., a speed of the moving object, using changes in the time base.
However, especially in the case where a large number of moving objects exist in one image, a process for correlating a plurality of objects in the time series images with each other for all the moving objects becomes difficult.
Further, when a plurality of stationary objects exist, as well as a plurality of moving objects, it becomes extremely difficult to rapidly complete such a correlation process for all of the stationary and moving objects using real time processing with a frame rate processing determined by a frequency of a video signal.
Furthermore, when a plurality of moving objects respectively move with a speed different from each other, it becomes almost impossible to complete the correlation process for all of the moving objects using real time processing determined by the frequency of a video signal (a video frame rate).
In view of the above-described problems existing in the prior art, the main object of the present invention is to provide an image processing apparatus which allows one or a plurality of objects to be rapidly and accurately extracted and analyzed, in a case where a large number of moving objects exist in a time series image.
A further object of the present invention is to provide an image processing apparatus which allows one or a plurality of objects to be rapidly and accurately extracted and analyzed, even in the case where a plurality of stationary objects exist, as well as a plurality of moving objects.
A still further object of the present invention is to provide an image processing apparatus which allows the movement of each of a plurality of moving objects to be rapidly and accurately extracted and analyzed, even in the case where the plurality of moving objects respectively move with different speeds.
A still further object of the present invention is to provide an image processing apparatus which allows all of the stationary and moving objects to be correlated with each other during real time processing with a processing rate determined by a frequency of a video signal, in the case where a plurality of stationary objects exist, as well as a plurality of moving objects, and also in a case where the plurality of moving objects respectively move with different speeds.
A still further object is to provide an image processing apparatus which allows a distance between two moving objects to be calculated, so that an abnormal object motion that may bring about an accident, a disaster, and the like, can be rapidly detected.
To attain these objects, the image processing apparatus according to the present invention includes an image-input unit which inputs an image including a background and a plurality of the objects; a background image extract unit which extracts the background; a first average background extract unit which extracts an image that includes one or a plurality of stationary objects or moving objects each having a speed not higher than a predetermined first speed and also includes the background; and a second average background extract unit which extracts an image that includes the stationary objects or moving objects each having a speed not higher than a predetermined second speed and also includes the background.
Further, the image processing apparatus of the present invention further includes a first difference-calculation processing unit which calculates a difference between an output from the background image extract unit and either one of outputs from the first average background extract unit, and then generates a first image containing objects moving at a first speed or stopped; a second difference-calculation processing unit which calculates a difference between respective outputs from the first and second average background extract units, and then generates a second image containing objects moving at a second speed; and a third difference-calculation processing unit which calculates a difference between an output from the image-input unit and either one of outputs from the first and second average background extract units, and then generates a third image containing objects moving at a third speed.
Preferably, the image processing apparatus of the present invention includes a plurality of local-area characteristic extract processing units which process outputs from the image-input unit. Each of the local-area characteristic extract processing units has a local-area determining unit which allocates the output from the image-input unit to each of a plurality of local areas; a labeling processing unit which separates at least one object from each of the local areas, by labeling the same object existing in each of the local areas; and a characteristic-amount calculating unit which calculates a plurality of characteristic-amounts or parameters, such as length and circumference, for the thus labeled object in the local areas.
Further, preferably, the image processing apparatus of the present invention operates to calculate a difference between the background and an average background image at a low speed, and to extract one or a plurality of connected areas where objects overlap.
Further, preferably, the image processing apparatus operates to produce a projection for each of the connected areas, and to calculate the position of the corresponding object in accordance with the projection, and to calculate a plurality of characteristics.
Further, preferably, the image processing apparatus operates to estimate a change in the position of the object and a change in the characteristics of the object for each sampling time period in the time series, and to determine whether the object is a stationary object, in a case where both the change in the position of the object and the change in the characteristics are small.
In a preferred embodiment, the image processing apparatus of the present invention is adapted to calculate a distance between two moving objects. The image processing apparatus includes an image-input unit which inputs the image including a background and a plurality of objects; a marker holding unit which places a plurality of markers in the background; a moving object extraction unit which extracts a plurality of moving objects; a tracing means which traces the plurality of moving objects; a marker extract unit which extracts the markers existing between the two different moving objects; and a distance measuring unit which calculates the distance between the moving objects, on the basis of the size of the extracted markers.
Further, in the image processing apparatus of a preferred embodiment, a plurality of other markers, which are not connected with each other by the marker holding unit, are provided in the background.
The image processing apparatus of a preferred embodiment further includes a connected-area position/shape calculating unit which calculates the size, the shape, and the number of the markers; a marker dictionary unit which has a marker dictionary for storing in advance the size and the shape of the markers; and a marker collating unit which collates the shape of the markers existing between two different moving objects and also collates the marker dictionary.
Further, the image processing apparatus of a preferred embodiment is adapted to calculate the number of the markers which can be identified as true markers based on a result of the collation in the marker collating unit, and to calculate the distance between two moving objects.
In a modified embodiment, the image processing apparatus of the present invention is adapted to calculate a distance between two cars in the case where a plurality of cars are the moving objects. In this case, a plurality of white lines are used as markers; these white lines are perpendicular to the direction in which the cars move with equal spaces between adjoining white lines.
Further, in this modified embodiment, the image processing apparatus has a distance measuring unit, which extracts the number of continuous white lines, and calculates the distance between two cars on the basis of the total sum of spaces between the continuous white lines between the two cars.
According to the image processing apparatus of the present invention, a plurality of objects existing in an image can be classified into a plural images each with objects of a different speed therein on the basis of the speed of each object, and the images can be analyzed in a process independent of each other.
Therefore, in the case where there are a large number of objects moving at various speeds, it becomes possible to separate objects in a certain classified range of speed from the remaining objects. Consequently, it becomes possible to easily and rapidly analyze the movement of only the objects within a certain range of speed.
Further, according to the image processing apparatus of the present invention, by extracting (or identifying) markers which can be easily processed by means of an image processing technique, portions where the moving objects and the markers overlap with each other can be easily determined, even though only a part of each moving object can be detected. Therefore, by calculating a distance between portions where the moving objects and the markers overlap with each other, it becomes possible to obtain a distance between two moving objects with a sufficiently high accuracy.